# Redis optimization

Retroactive User Recognition is using Redis to store all anonymous incoming events. Usually,
the volume of such event can be pretty huge, so Redis can take a lot of RAM (note: Redis keeps
all data in RAM, disk is used only for persistence)

Approximate of event in Redis is `1KB`. So if the project processes
requests per second, and let's say 80% of them are anonymous, the daily consumption will be:

```
8 events * 86,400 (seconds per day) = ~700Mb
```

By default, Jitsu will mark users not active for 7 days for eviction, so total Redis consumption will be around ~5G
(number of inactivity days is controlled by `USER_RECOGNITION_TTL_MINUES`)

Redis keys for user recognition format is `anonymous_events:destination_id#${destinationID}:anonymous_id#${userAnonymousID}`.
Example: `anonymous_events:destination_id#my_postgres:anonymous_id#bqsyuwusxh`. All events generated by this user
will be stored under this key is list

The most reliable way to handle the RAM problem is having a dedicated Redis instance for user recognition with memory limit and [keys eviction](https://redis.io/topics/lru-cache) configuration.
You can configure it with the following steps:

### 1. Configure separate Redis in Jitsu Server configuration file (eventnative.yaml) with anonymous events TTL (default value is 7 days):

```yaml
users_recognition:
  enabled: true
  identification_nodes:
    - /user/id
    - /user/email
  redis:
    host: redis_host
    port: 6379
    password: secret_password
    ttl_minutes: #Optional. Anonymous events TTL in minutes. Default value: 10080 (7 days)
      anonymous_events: 10080 #7 days
```
Alternatively, you can configure dedicated redis with env variable (`USER_RECOGNITION_REDIS_URL=redis://:password@host:port`) and TTL with
`USER_RECOGNITION_TTL_MINUTES`

TTL (time to live) is applied to Redis keys. It means that events from anonymous users that no longer visit your web pages will be deleted from Redis at the expiration time.
Under the hood Jitsu uses [expire command](https://redis.io/commands/expire) after saving every anonymous event JSON.
Since Redis anonymous events key contains user anonymous ID, Redis will automatically delete key if there was no events from a certain anonymous user 7 days (TTL value).


### 2. Specify Redis memory limit and eviction policy by adding the following configuration into the `redis.conf` file:

```
maxmemory 25gb
maxmemory-policy allkeys-lru
maxmemory-samples 5
```

Or use command `CONFIG SET` with `redis-cli`:

```
CONFIG SET maxmemory 25gb
CONFIG SET maxmemory-policy allkeys-lru
CONFIG SET maxmemory-samples 5
```

<Hint>
    <code inline="true">CONFIG SET</code> command change current Redis configuration, but doesn't persist the configuration between server restarts. Make sure that you have added new values to the configuration file after using <code inline="true">CONFIG SET</code> command.
</Hint>

After this configuration, user recognition Redis will automatically rewrite older keys when RAM is almost full.

Add Redis compression configuration, so all Redis hash tables with length > 1 (anonymous with more than 1 event inside 1 key) will be converted to the [Redis ziplists](https://redis.com/ebook/part-2-core-concepts/01chapter-9-reducing-memory-use/9-1-short-structures/9-1-1-the-ziplist-representation/)
and compressed. Compression rate is about 8x-10x. It saves about 800-1000% of RAM.

<Hint>
  <ul>
    <li><b>Compression isn't applied to old data</b>. Compression applies only to Redis hash tables which have been created after this configuration.</li>
    <li><b>Compression applies only to hash tables with length > 1</b>. It means that size of a hash table with 1 event =~ 1KB, and size with hash table with 2 events =~1.08KB, 3 events =~1.16KB.</li>
  </ul>
</Hint>

Configuration:

```
list-compress-depth 1
hash-max-ziplist-value 2000
```

Or use command `CONFIG SET` with `redis-cli`:

```
CONFIG SET list-compress-depth 1
CONFIG SET hash-max-ziplist-value 2000
```

You can check how much memory any key consumes with `redis-cli` command:

```
DEBUG OBJECT <REDIS KEY>
```

If you can't change the redis configuration, you can enable GZIP compression on the Jitsu side. It will approximately reduce Redis memory consumption by 30%.
Just add `users_recognition.compression: 'gzip'` to the Jitsu Server configuration (eventnative.yaml):

```yaml
users_recognition:
  enabled: true
  compression: 'gzip'
  identification_nodes:
    - /user/id
    - /user/email
  redis:
    host: redis_host
    port: 6379
    password: secret_password
    ttl_minutes: #Optional. Anonymous events TTL in minutes. Default value: 10080 (7 days)
      anonymous_events: 10080 #7 days
```

<Hint>
    We do not recommend use Redis ziplist compression and Jitsu GZIP compression at the same time
</Hint>